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ABSTRACT 

The field of formal evaluation is a newly cl^eloping 
one which /requires that training programs encompass new concepts as 
well as those that ar^ standard in preparing professionals for the 
field. They should be I trained as data gatherers in many roles, they 
should be aware of new developments and be able to contribute to 
them. The curriculum make up of three universities, whom the author 
sees as having successful training programs are examined* 
Considerations are made as to what factors formal evaluation training 
programs should meet., such as the differences in training for 
academicians and a prof essional evaluators, the content of the 
program, and the personality types of students selected for these 
programs, (Author/DEP) 
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\fliat are we doing in formal evaluation training programs? What (in 
the world) are we doing in formal evaluation training programs? Depending 
on the inflection used in the tit],e, the content of this paper could take 
different directions. My intention is to briefly examine the make-up of 
some of the more successful evaluation training programs and then to 
describe some of their deficiencies as I see them. Finally, I want to 
propose a new set of prescriptions based partially on logic, partially on 
experience, and heavily on intuition. I ^uspe t that not everything I say 
should be accepted as "the truth" and I hope this paper will serve to 
stimulate dialogue around at least some points* 

Since I have written this paper without the benefit of reviewing the 
presentations my colleagues prepared for this symposium, I suspect there 
may be some overlap with topics they are addressing. I hope the overlap 
is minimal. suspect, too, that some of what I say will be in conflict 
with their viewpoints. Yet in the interest of stimulating discussion, I 
hope the conflict is maximal — at least in certain areas. Because of time 
restrictions, I cannot cover all themes in this paper as comprehensively 
rs I would wish, but I am prepared to discuss each topic further, and hope 
we have an opportunity to do so. 

What are we doing in foimal evaluation training progr^s? 

A few successful evaluation training programs currently exist in our 
institutions of higher learning. In judging success, I consider the 
productivity and Impact of program graduates in their chosen evaluation 
endeavors. By training I mean the preparation of persons to either practice 
or conduct research in educational evaluation. Generally, training can 
be categorized as either preservice or inservice, I will discuss only 
formal preservice training programs. Three such programs of which I am 
aware are the University of Colorado's Laboratory of Educational Research 
(LER), the University of Illinois' Center for Instructional Research and 
Curriculum Evaluation (CIRCE), and the Ohio State University's (recently 
transformed into Western Michigan University's) Evaluation Center ♦ Consider 
what each is doing. 

LER has a structured set of basic courses in statistics and research 
design (at least four courses), measurement (at least one course), and 
evaluation (usually one course). LER concentrates on turning out 
methodologists who are prepared for university research and/or teaching 



A paper prepared for presentation in a symposium, The Traininy^ , Care , and 
Feedin j ^ of Educational Evaluators , Annual Meeting of the American Bducationa! 
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wishes to thank Drs. Nick L. Smith and David Churchmen for sharing their 
ideas on the formal training of educational evaluators. Responsibility for 
the content of this paper, however, remains in the hands of the author. 



positions. Internships with an emphasis on research and methodological 
consultation are required of all students from the time they enter 
the program, LER faculty are conducting quality research themselves 
and are highly visible nationally. Selection of students for the 
program is structured and includes consideration of the student's past 
performance in coursework, research and/or evaluation experience, as 
well as quantitative aptitude and recommendations. Faculty-student 
interactions on professional topics outside the classroon are frequent 
(stimulated at least in part by weekly brown bag seminars). 

The CIRCE program includes a structured set of basic courses in 
statistics (at least one course) > measurement (at least one course), and 
evaluation (usually one course). These courses are designed to produce 
content matter specialists (methodologists, economists, journalists, 
curriculum specialists, lawyers, etc.) who can apply and are interested 
in applying their specialized knowledge to educational evaluation. 
Consulting arrangements involving the faculty are shared with CIRCE 
students at every opportunity. Apprenticeship arrangements with units 
throughout the university are established according, to students* 
interests. The CIRCE faculty are involved in quality research and 
evaluation activities and are highly visible nationally. Selection of 
students for the program is unstructured. Typically a student's interest 
in CIRCE is enough to draw him into the fold. Outside of a weekly brown 
sack seminar and the basic courses already mentioned, the program is not 
highly structured. The student's interest and university selection and 
graduation requirements set the direction for participants in the program. 

The Evaluation Center's training program has basic courses in 
statistic^ (at least one course) , measurement (at least one course) , 
and evaluation (at least one course). A student's program is typically 
defined in consultation with his advisor, who may or may not be a member 
of the Evaluation Center staff. For the most part, graduates of the 
Center are prepared to take positions in evaluation. Both masters and 
doctoral level students are graduated from the program. Students in the 
, program are employed as full- or part-time staff members of the Center* 
I As such, they are involved in evaluation or evaluation-related work 
; commensurate with their interests and level of training. The faculty 
at the Center teach part time and regularly engage In quality research 
and evaluation. They are highly visible nationally. Selection of students 
for the program is structured and Inclm'.es a recruitment component that 
permits self-selection even before faculty are contacted. Because 
students are actually employees of the Center, frequent faculty-student 
Interactions occur on project work outside the classroom. Students are often 
given professional responsibilities similar to those given full-time 
practicing evaluation project directors. 

What are the reoccurring characteristics of evaluation training 
programs at these three institutions? Bucknell's Bill Moore and Hugh 
McKeegan summarized it well a few years ago when they stated: 

...quality staff, quality laboratories, quality 
programs, .. .quality students. . .are necessary 
conditions for any quality. . .program. ^ 



''J, William Moore and Hugh F. McKeegan. An Emphasis in Educational 
Research for the Undergraduate 'Student. Paper presented at 1970 AERA 
Annual Meeting. 




Current work in evaluation training is being augmented by researchers 
interested in prescribing directions for developing new evaluation 
training programs. The AERA Task Force on Research Training has yielded 
an impressive series of tcchmical papers and a list of competencies nejdel 
in educational research and evaluation. 3 The Oregon studies^ have yielded 
a similar list; Coller (1970)5 a similar list, Sanders (1970) etc. 
The outcome statements reflected in these works provide the evaluation 
trainer and trainee with a target that many feel is worth aiming for. 
Strategies for hitting that target, suggested by a glimpse at the more 
successful evaluation training programs, include: 

1. Providing basic coursework in statistics, 
measurement and evaluation 

2. Selecting students interested in evaluation 

3. Providing supervised field experiences for students 

4. Providing faculty actively involved in quality 
research and evaluation 

5. Providing faculty who are nationally visible 
in educational evaluation 

6. Offering frequent, out-of-classroom faculty-student 
professional interactions 

Wh.t fin the worlds are we doing in forma] Pv.ltiation traininfi programs? 

Is what we are doing sufficient? Is it adequate? I think not. 
First, consider the roles that evaluation trainees are asked to assume. 
Generally, there are-^^categories of roles for evaluators, each of which 
enco^ass;s a number of secondary roles. They are: P^^^ 
an5 Z) academician. The professional is a technician who conducts one 
evaluation study after another. The academician is a 

applies his special skills to improve the practice of ^'1"^^'^^°"^^^^^;^'" 
These roles a^e not mutually exclusive; the academician should have all 
the skills of the professional and be able to function in that role. 



3see, for example, B. R. Worthen, "Competencies for Educational Research 
and Evaluation," Educational Researcher, 197^, 4, IJ-lb. 

^H. D. Schalock and G. R. Sell. The Oregon studies in educational 
research, development, diffusion, and evaluation; Volume I, Summary Report 
(with technical appendices). Monmouth, Oregon, Teaching Research Division 
Oregon State System of Higher Education, 1972. 

5a. R. CoUer. A taxonomy of programmatic tasks in J^J^f "J''^'^^"!;^^ . ^. 
evaluation, facilitation and coordination system. Northfield, Illinois. 
Cooperative Educational Pesearch Laboratory, Inc., IV/U. 

R Sanders. Evaluation training design. In R. L. Turner, £^^- _ 
New Patterns for Training R, D, D, and E Personnel. Bloomxngton, Indiana. 
Indiana University, 1970. 
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With the exception of the Evaluation Center, few professionals are 
produced by evaluation training programs. Doctoral program graduates 
are often nominally prepared for that role, but their interests in 
conducting research or working in their speciality areas (if not - research) 
quickly entice them away from the professional role. The current need 
to prepare undergraduate and masters level students for the professional 
role is rtot being adequately met. The training programs for these 
people should provide them with the basic knowledge of statistics, 
measurement, and evaluation they need to conduct evaluation studies under 
the direction of the academician. This is not a new idea; it has been 
expressed before. But the need for action remains. 

Second, consider the cont^xlL in which evaluation trainees must work. 
Sanders and Cuba (1973)^ identified and described a list of nine factors 
that may (and probably will) influence the quality of an evaluation study: 



1. 


Potential use 


2. 


Audience 


3. 


Methodological 


4. 


Cost 


5. 


Human 


6. 


Knowledge 


7. 


Organizational 


8. 


Legal 


9. 


Political 



VThat formal training prepares evaluation trainees to deal with these 
factors? Wl-.at experiences should be provided to students of evaluation 
to prepare them for "the real world?" Certainly work experience is 
relevant. But, can the in-again, out-again type of consulting, conducted 
by university faculty members or short-term project directors adequately 
communicate the nature of these factors— or instill in students the necessary 
sensitivity to these factors? If past experience is any indication, the 
answer is no. Is course work the answer? Not for evaluation trainees. 

What, then, can be done that is not already being tried? I propose 
that a minimum of six months in every professional's or academician s 
program be set aside for on-site work in evaluation at locations offering 
a good mixture of t^ - factors listed above. Regional Educational 
Laboratories and State Departments of Education are two appropriate 
locations that come immediately to mind. At Northwest Regional Educational 
Laboratory we have tried this arrangement with evaluation trainees from 



7j. R. Sanders and E. C Cuba. A taxonomy of problems confronting the 
practitioner of educational evaluation. Bloomington, Indiana: Indiana 
University, 1973. 



Brigham Young University; it has met with considerable success. We are 
learning to effectively structure apprenticeship experiences. 

How should these experiences be structured? An apprentice should 
be deeply involved in at least one project. He should live with and 
observe the day-to-day headaches and decisions; he should be given 
responsibilities ^so that when mistakes are made, he has to live with them. 
He should be put into a position such that when questions about appropriate 
content or strategies for evaluation arise, he alone is forced either to 
find answers or to find gaps in our knowledge that call for research. 
Naturally this responsibility cannot be assigned on the first day of work. 
I suggest, however, that a sequence of experiences can be structured so 
that a student advances through progressively demanding stages of 
responsibility. Such advancement can be enacted only through long-term, 
full-time work experience. 

Third, consider the content of formal evaluation training. Content 
is related to roles for which evaluation trainees are being prepared, but 
there are some unique considerations related to the content of instruction. 
Worthen's (1975) list of competencies represents an ideal. Although I 
genuinely hate to protest a nice number like 25, I suspect this list is 
incomplete (viz., notice the contextual factors suggested by Sanders and 
Cuba that I suggest should be dealt with by practicing evaluators). A more 
pressing concern, however, is that this list may unwittingly lead 
evaluation trainers to conclude that formal training is the only way to 
impart these competencies, and that every evalrator should be able to 
function in each competency area. This concern remains even though authors 
halve carefully prepared disclaimers for such lists. 



/ Consider training for the professional . Certainly training in the 
bl^sic skills^ of measurement, data analysis, and evaluation planning is 
es^ntlal for the student who does not already have those skills. 
Selection of students for professional training programs is also important. 
Personological variables such as independence, interpersonal skills, 
ability to cope with political and other factors that affect evaluations, 
flexible content interest, ability to communicate, task orientation and 
interest in functioning in the technical (as opposed to the research) role 
are not easily imparted to a student. Therefore, a student entering the 
program should already demonstrate these characteristics. Exp erience is 
also a vital component of professional preparation. The apprenticeship 
arrangement just discussed offers ideal training for professionals through 
experience. Finally, the b^ackground of potential trainees should be 
considered. A history of productivity on the job (regardless of the 
nature of the task), elective courses and/or interest in educationally or 
socially related subjects, and successful experiences in dealing with 
people are all important. 

Training for the yademician should differ. Basic applied and 
conceptual skills in measurement, data analysis (including computer 
application), evaluation planning, and project management are essential. 
A good dose of philosophy of science would also be valuable. This person, 
I would argue, should not be trained as a specialist in evaluation, however* 
Rather, his training should be in a substantive area, such as economics. 
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political science, sociology, quantitative methodology, law, philosophy, 
and so on. This preparation would enable him to function at the 
university level in his chosen field. He could then apply bis 
substantive knowledge in studying evaluation as a subject of inquiry. 
Selection of academicians is as important as selection of professional 
trainees. The academician should have a genuine interest in studying 
and improving educational evaluation. He should be interested in applying 
his conceptual skills to direct the work of others. Like the professional, 
he should be intelligent, be able to work independently, have high level 
interpersonal skills, and be able to cope with the contextual constraints 
of evaluation (although his interest would lie in objective treatment 
of those constraints as conditions v7orthy of study}v His experience 
should follow the apprenticeship approach described earlier, but 
encompass a higher level of conceptual work and managerial responsibility. 
The products of this experience would of course differ from those of the 
professional; the academician would be expected to produce conceptual and 
research papers, and management (including budget) plans, and to demonstrate 
the successful direction of entire evaluation projects. Finally, the 
background of the academician should reflect productivity in conceptual 
and^ research work in some substantive area.. 

A fourth, and last, consideration in training evaluators is 
related to the chanRing face of the field. Conceptually, it is a young 
area of study. Technically it is old. Professionals should be trained 
to function as data gatherers in many roles; academicians should be 
trained in specialty areas outside of evaluation, yet maintain strong 
interest in conducting evaluation research. Then, as the face of evaluation 
changes — as it is certain to do — job security will not be a concern • 
Evaluation trainees should be aware of new developments and be prepared to 
contribute to them as they appear. Academicians in particular should be 
prepared to critique new directions in educational evaluation, knowing 
that the area is still growing and changing but recognizing that the 
change only increases the need for direction. 



A text on educational evaluation recently crossed my desk. On 
the inside leaf of the cover the authors had stated, "This new book 
contains everything professional educators need to know about program 
evaluation . . At last, I thought , here is an evaluation training program 
is one nice, neat package. My expectations were not met. 
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